shortIRT Package\(^1\) University of Padova, Padova \(^2\) Psicostat, Padova \(^3\) University of Trento, Rovereto
Many items \(\rightarrow\) good measurement precision, great reliability and so on
Not always!
People might get tired and frustrated
IRT models for the win
Being focused on the item information and on the ability of each item to measure different levels of the latent trait, IRT models provide an ideal framework for developing STF (and not torturing people)
Different approaches: Automated test assembly procedures
According to the 4-Parameter Logistic Model:
\[P(x_{pi}=1|\theta_p, b_i, a_i, c_i, e_i) = P(\theta) = c_i + (e_i-c_i)\dfrac{\exp[a_i(\theta_p - b_i)]}{1 + \exp[a_i(\theta_p - b_i)]}\]
Item Characteristics Curves (ICCs)
\[ \text{IIF}_{i}(\theta) = \dfrac{a_i^2[P(\theta)-c_i]^2[e_i - P(\theta)]^2}{(e_{i}-c_i)^2 P(\theta)[1-P(\theta)]}\]
\[TIF(\theta) = \sum_{i = 1}^{|B|} IIF_i(\theta)\] (\(B\): Set of items in a test (\(|X|\) cardinality of set \(X\)))
Create a short test form composed of \(N\) items from an item bank \(B\) \(\rightarrow\) Select the \(N\) items with the highest IIFs:
The IIFs of the items of item bank are sorted in decreasing order:
\[\mathit{iif} = (\displaystyle \max_{1 < i < B} IIF_i(\theta), \ldots \displaystyle, \min_{1 < i < B} IIF_i(\theta)) \]
Items with IIFs from 1 to \(N\), \(N < ||B||\), are selected to be included in the short test form
Aim: STF with \(N = 3\) items from a full-length test of 10 items:
| Item | \( b_i \) | \( a_i \) | \( c_i \) | \( e_i \) | \( \text{IIF}_i(\theta) \) |
|---|---|---|---|---|---|
| 1 | 1.65 | 1.32 | 0.10 | 1 | 0.36 |
| 2 | -1.82 | 0.71 | 0.06 | 1 | 0.11 |
| 3 | 2.87 | 0.78 | 0.00 | 1 | 0.15 |
| 4 | -1.79 | 0.81 | 0.01 | 1 | 0.16 |
| 5 | -0.83 | 0.87 | 0.08 | 1 | 0.16 |
| 6 | 1.46 | 1.35 | 0.03 | 1 | 0.43 |
| 7 | 2.87 | 0.73 | 0.00 | 1 | 0.13 |
| 8 | -0.01 | 1.41 | 0.06 | 1 | 0.44 |
| 9 | -2.92 | 1.09 | 0.06 | 1 | 0.26 |
| 10 | -1.44 | 1.07 | 0.09 | 1 | 0.24 |
Aim: STF with \(N = 3\) items from a full-length test of 10 items:
| Item | \( b_i \) | \( a_i \) | \( c_i \) | \( e_i \) | \( \text{IIF}_i(\theta) \) |
|---|---|---|---|---|---|
| 8 | -0.01 | 1.41 | 0.06 | 1 | 0.44 |
| 6 | 1.46 | 1.35 | 0.03 | 1 | 0.43 |
| 1 | 1.65 | 1.32 | 0.10 | 1 | 0.36 |
| 9 | -2.92 | 1.09 | 0.06 | 1 | 0.26 |
| 10 | -1.44 | 1.07 | 0.09 | 1 | 0.24 |
| 4 | -1.79 | 0.81 | 0.01 | 1 | 0.16 |
| 5 | -0.83 | 0.87 | 0.08 | 1 | 0.16 |
| 3 | 2.87 | 0.78 | 0.00 | 1 | 0.15 |
| 7 | 2.87 | 0.73 | 0.00 | 1 | 0.13 |
| 2 | -1.82 | 0.71 | 0.06 | 1 | 0.11 |
Aim: STF with \(N = 3\) items from a full-length test of 10 items:
| Item | \( b_i \) | \( a_i \) | \( c_i \) | \( e_i \) | \( \text{IIF}_i(\theta) \) |
|---|---|---|---|---|---|
| 8 | -0.01 | 1.41 | 0.06 | 1 | 0.44 |
| 6 | 1.46 | 1.35 | 0.03 | 1 | 0.43 |
| 1 | 1.65 | 1.32 | 0.10 | 1 | 0.36 |
| 9 | -2.92 | 1.09 | 0.06 | 1 | 0.26 |
| 10 | -1.44 | 1.07 | 0.09 | 1 | 0.24 |
| 4 | -1.79 | 0.81 | 0.01 | 1 | 0.16 |
| 5 | -0.83 | 0.87 | 0.08 | 1 | 0.16 |
| 3 | 2.87 | 0.78 | 0.00 | 1 | 0.15 |
| 7 | 2.87 | 0.73 | 0.00 | 1 | 0.13 |
| 2 | -1.82 | 0.71 | 0.06 | 1 | 0.11 |
\(k = 0, \ldots, K\): Scalar denoting the iterations of the procedures (\(K = N-1\))
\(S^k \subseteq \{1, \ldots, J\}\): Set of items selected to be included in the short test form up to iteration \(k\)
\(Q^k \subseteq \{1, \ldots, N\}\): Set of \(\theta'\)s satisfied up to iteration \(k\);
At \(k=0\): \(S^0 = \emptyset\), \(Q^0 = \emptyset\)
The procedure cycles steps 1 to 3 until \(k = K\):
At iteration \(K\), \(|Q^{K + 1}| = N\) and \(|S^{K + 1}| = N\)
\(\theta'\)
|
|||||||
|---|---|---|---|---|---|---|---|
| 1 | 2 | \(\ldots\) | n | \(\ldots\) | N | ||
| 1 | \(\mathit{iif}_{11}\) | \(iif_{12}\) | \(\vdots\) | ||||
| 2 | \(\mathit{iif}_{21}\) | \(\mathit{iif}_{22}\) | \(\vdots\) | ||||
| \(\vdots\) | \(\vdots\) | ||||||
| \(i\) | \(\ldots\) | \(\ldots\) | \(\ldots\) | \(\mathit{iif}_{in}\) | \(\ldots\) | \(\ldots\) | \(\ldots\) |
| \(\vdots\) | \(\vdots\) | ||||||
| \(B\) | \(\vdots\) | \(\mathit{iif}_{BN}\) | |||||
Aim: Develop a STF of \(N=3\) items from \(B\) with \(\theta' = (-2,0,2)\):
| Item | b | a | c | e |
|---|---|---|---|---|
| 1 | 1.65 | 1.32 | 0.10 | 1 |
| 2 | -1.82 | 0.71 | 0.06 | 1 |
| 3 | 2.87 | 0.78 | 0.00 | 1 |
| 4 | -1.79 | 0.81 | 0.01 | 1 |
| 5 | -0.83 | 0.87 | 0.08 | 1 |
| 6 | 1.46 | 1.35 | 0.03 | 1 |
| 7 | 2.87 | 0.73 | 0.00 | 1 |
| 8 | -0.01 | 1.41 | 0.06 | 1 |
| 9 | -2.92 | 1.09 | 0.06 | 1 |
| 10 | -1.44 | 1.07 | 0.09 | 1 |
| 0.00 | 0.08 | 0.35 |
| 0.11 | 0.08 | 0.03 |
| 0.01 | 0.05 | 0.13 |
| 0.16 | 0.10 | 0.03 |
| 0.11 | 0.15 | 0.05 |
| 0.00 | 0.16 | 0.38 |
| 0.01 | 0.05 | 0.12 |
| 0.05 | 0.44 | 0.10 |
| 0.21 | 0.04 | 0.01 |
| 0.21 | 0.15 | 0.02 |
\(S^0 = \emptyset\)
\(Q^0 = \emptyset\)
| 0 | 0.08 | 0.35 |
| 0.11 | 0.08 | 0.03 |
| 0.01 | 0.05 | 0.13 |
| 0.16 | 0.1 | 0.03 |
| 0.11 | 0.15 | 0.05 |
| 0 | 0.16 | 0.38 |
| 0.01 | 0.05 | 0.12 |
| 0.05 | 0.44 | 0.1 |
| 0.21 | 0.04 | 0.01 |
| 0.21 | 0.15 | 0.02 |
\(\mathit{iif}_{\text{max}}^0=\displaystyle \max_{j \in J\setminus S^0, \, n \in N \setminus Q^0} \mathbf{IIF}= \mathbf{IIF}(8,2) = 0.44\)
\(S^{1} = S^0 \cup \{8\}\) = {8}
\(Q^{1} = Q^0 \cup \{3\}\) = {2}
| 0 | 0.08 | 0.35 |
| 0.11 | 0.08 | 0.03 |
| 0.01 | 0.05 | 0.13 |
| 0.16 | 0.1 | 0.03 |
| 0.11 | 0.15 | 0.05 |
| 0 | 0.16 | 0.38 |
| 0.01 | 0.05 | 0.12 |
| 0.05 | 0.44 | 0.1 |
| 0.21 | 0.04 | 0.01 |
| 0.21 | 0.15 | 0.02 |
\(\mathit{iif}_{max}^1=\displaystyle \max_{j \in J\setminus S^1, \, n \in N \setminus Q^1} \mathbf{IIF} = \mathbf{IIF}(6,3)= 0.38\)
\(S^{2} = S^1 \cup \{6\} = \{8, 6\}\)
\(Q^{2} = Q^1 \cup \{3\} = \{2, 3\}\)
| 0 | 0.08 | 0.35 |
| 0.11 | 0.08 | 0.03 |
| 0.01 | 0.05 | 0.13 |
| 0.16 | 0.1 | 0.03 |
| 0.11 | 0.15 | 0.05 |
| 0 | 0.16 | 0.38 |
| 0.01 | 0.05 | 0.12 |
| 0.05 | 0.44 | 0.1 |
| 0.21 | 0.04 | 0.01 |
| 0.21 | 0.15 | 0.02 |
\(\mathit{iif}_{max}^2=\displaystyle \max_{j \in J\setminus S^1, \, n \in N \setminus Q^1} \mathbf{IIF} = \mathbf{IIF}(9,1)= 0.21\)
\(S^{3} = S^2 \cup \{9\} = \{8, 6, 9\}\)
\(Q^{3} = Q^2 \cup \{1\} = \{2,3, 1\}\)
| 0 | 0.08 | 0.35 |
| 0.11 | 0.08 | 0.03 |
| 0.01 | 0.05 | 0.13 |
| 0.16 | 0.1 | 0.03 |
| 0.11 | 0.15 | 0.05 |
| 0 | 0.16 | 0.38 |
| 0.01 | 0.05 | 0.12 |
| 0.05 | 0.44 | 0.1 |
| 0.21 | 0.04 | 0.01 |
| 0.21 | 0.15 | 0.02 |
\(|S^3| = 3\), \(|Q^3| = 3\), \(K = 2\) \(\rightarrow\) end
shortIRT packageIt’s on CRAN!
bench()bench(item_par, iifs = NULL, theta = NULL, num_item = NULL)
set.seed(1312)
n = 10
item_par = data.frame(b = runif(n, -3,3),
a = runif(n, .7, 1.5),
c = runif(n, 0, .10),
e = 1)
theta = rnorm(1000)
stf = bench(item_par, theta = theta, num_item = 3)bench()bench(item_par, iifs = NULL, theta = NULL, num_item = NULL)
set.seed(1312)
n = 10
item_par = data.frame(b = runif(n, -3,3),
a = runif(n, .7, 1.5),
c = runif(n, 0, .10),
e = 1)
theta = rnorm(1000)
stf = bench(item_par, theta = theta, num_item = 3)
summary(stf)
plot(stf) + theme_bw()The selected items are
10 2 9
These items maximize the information for thetas equal to:
2.251 -1.215 -1.077
The parameters of the selected items are:
b a c e
10 2.243866 1.390367 0.02130947 1
2 -1.246596 1.283032 0.02087452 1
9 -1.171959 1.344295 0.07825710 1
The item selection is based on the bench procedure
define_targets()define_targets(theta, num_targets = NULL, method = c("equal", "clusters"))
theta_target()theta_target(targets, item_par)
The selected items are
2 6 8
These items maximize the information for thetas equal to:
-1.207019 1.213158 0.01562251
with the following parameters
b a c e
2 -1.2465959 1.283032 0.02087452 1
6 0.8283301 1.198013 0.01028904 1
8 0.3331977 1.208366 0.09370018 1
The item selection is based on the theta-target procedure with clusters target
The selected items are
10 8 2
These items maximize the information for thetas equal to:
2.485 0.045 -2.395
with the following parameters
b a c e
10 2.2438664 1.390367 0.02130947 1
8 0.3331977 1.208366 0.09370018 1
2 -1.2465959 1.283032 0.02087452 1
The item selection is based on the theta-target procedure with equal target
The selected items are
10 6 4
These items maximize the information for thetas equal to:
2 2 2
with the following parameters
b a c e
10 2.2438664 1.3903672 0.02130947 1
6 0.8283301 1.1980125 0.01028904 1
4 0.7957328 0.9924159 0.08716286 1
The item selection is based on the theta-target procedure with unknown target
Boh funziona si possono dire tante cose ma non so cosa dire
Psychoco 2026 - Padova